Skip to content

Validate generation batch limits - #1681

Draft
Thump604 wants to merge 1 commit into
ml-explore:mainfrom
Thump604:604/upstream-generation-validation
Draft

Validate generation batch limits#1681
Thump604 wants to merge 1 commit into
ml-explore:mainfrom
Thump604:604/upstream-generation-validation

Conversation

@Thump604

@Thump604 Thump604 commented Aug 9, 2026

Copy link
Copy Markdown

What changed

  • Return an empty stream for max_tokens=0 and reject negative limits.
  • Validate every per-prompt option list before creating caches or mutating batch state.

Root cause

Zero-token generation reached final response construction with uninitialized loop variables. Batch insertion used unchecked zip, silently truncating prompts when any option list was short.

Validation

  • Focused zero-token and short-list regressions pass.

Fixes #1689

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch generation can silently omit prompts and mishandles max_tokens=0

1 participant